Spatial and temporal multiplexing display

ABSTRACT

Described is using a combination of which a multi-view display is provided by a combining spatial multiplexing (e.g., using a parallax barrier or lenslet), and temporal multiplexing (e.g., using a directed backlight). A scheduling algorithm generates different views by determining which light sources are illuminated at a particular time. Via the temporal multiplexing, different views may be in the same spatial viewing angle (spatial zone). Two of the views may correspond to two eyes of a person, with different video data sent to each eye to provide an autostereoscopic display for that person. Eye (head) tracking may be used to move the view or views with a person as that person moves.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is related to the copending U.S. patentapplication 12/819,239, entitled “Optimization of a Multi-View Display,”assigned to the assignee of the present application, filed concurrentlyherewith and hereby incorporated by reference.

BACKGROUND

Display technology is evolving, including to the point where generalconsumers today can purchase 3D (three-dimensional/stereoscopic)televisions. In general, 3D televisions display separate video frames toeach eye via 3D glasses with lenses that block certain frames and passother frames through.

Spatial multiplexing is another recent advancement, in which alenticular display uses lenses (parallax barriers/lenslets) in front oflight sources (e.g., light emitting diodes/LEDs or liquid crystaldisplays/LCDs) to display separate views to different viewing angles(zones). The highest lenticular display, for example, enables nineseparate views, whereby viewers of the display who are positioned atappropriate angles (e.g., zones ten degrees apart) can each see adifferent image at the same time.

However, spatial multiplexing comes at the cost of resolution, in thatnine vertical columns of pixels on an LCD display, for example, areneeded to produce a single effective column of pixels for each viewingangle in the nine-view display. Given the limited resolution of LCDdisplays, this one-ninth degradation of effective horizontal resolutionis dramatic, particularly when considering that the effective verticalresolution is unchanged (and therefore significantly mismatched with thehorizontal resolution).

SUMMARY

This Summary is provided to introduce a selection of representativeconcepts in a simplified form that are further described below in theDetailed Description. This Summary is not intended to identify keyfeatures or essential features of the claimed subject matter, nor is itintended to be used in any way that would limit the scope of the claimedsubject matter.

Briefly, various aspects of the subject matter described herein aredirected towards a technology by which a multi-view display is providedby combining spatial multiplexing (e.g., using a parallax barrier orlenslet), and temporal multiplexing (e.g., using a directed backlight).A scheduling algorithm generates a plurality of views by determiningwhich light sources are illuminated at a particular time. Two of theviews may correspond to two eyes of a person, with different video datasent to each eye to provide an autostereoscopic display for that person.Eye (head) tracking may be used to move the view or views with a personas that person moves.

In one aspect, positions of a first view and a second view are used todetermine subsets of light sources (e.g., pixel columns) of a displaythat, when illuminated behind a lens, direct light to a spatial viewingangle (zone) corresponding to the both the first view and the secondview. In a first time slice, selected light sources of the subsets areilluminated to direct collimated light corresponding to first video datato the first view. In a second time slice, other selected light sourcesof the subsets are illuminated to direct collimated light correspondingto second video data to the second view. In this manner, for example,two or more views may be displayed within the same spatial viewingangle.

Other advantages may become apparent from the following detaileddescription when taken in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example and not limitedin the accompanying figures in which like reference numerals indicatesimilar elements and in which:

FIG. 1 is a representation of a directed backlight technology for use intemporal and spatial multiplexing, in which a directed backlightcollimates light generated by one or more light sources.

FIG. 2 is a block diagram representing a temporal and spatialmultiplexing system that generates separate views.

FIG. 3 is a flow diagram representing example steps for computing viewsbased on various input including a number of views desired.

DETAILED DESCRIPTION

Various aspects of the technology described herein are generallydirected towards combining temporal and spatial multiplexing to achievedesired video output, such as generating more views than eithertechnique can deliver independently. In one aspect, spatial multiplexing(using a parallax barrier or lenslet) provides different views todifferent viewing angles, while temporal multiplexing (e.g., via acollimated directed backlight) directs views based on illuminating onlyselected light sources at any given time, e.g., at a sufficiently highfrequency that flicker is not detected by a viewer. When used with knownhead tracking technology that determines the position of a viewer'seyes, the combined temporal and spatial multiplexing can provide eachperson (or each individual eye of a person) with a separate view in bothspace and time, thereby providing a multiplicative effect as to thenumber of individual views possible.

It should be understood that any of the examples herein arenon-limiting. As such, the present invention is not limited to anyparticular embodiments, aspects, concepts, structures, functionalitiesor examples described herein. Rather, any of the embodiments, aspects,concepts, structures, functionalities or examples described herein arenon-limiting, and the present invention may be used in various ways thatprovide benefits and advantages in computing and video technology ingeneral.

FIG. 1 represents how a time-multiplexed collimated directed backlight'sarrayed light sources (e.g., LEDs or LCDs represented by the smallcircles in FIG. 1) are aligned with lenses 102-104 to output differentphotons to different viewing angles based on which light source isilluminated (shown as a white circle) at any time. In general, adirected backlight collimates light generated by one or more lightsources, in which the angle of the collimated light depends on theposition of the currently illuminated one or more LEDs or LCDs withrespect to the focal point of the lens. As can be readily appreciated,in a practical display there are millions of such light sources, and anyone-dimensional or two-dimensional array of such light sources may beused, as appropriate for any shape of lens, e.g., convex, cylindrical,spherical and so forth.

In general and as described herein, software and hardware using acombination of spatial and temporal multiplexing techniques control thedrivers 106 to illuminate only selected light sources during a giventime slice, for each of the arrays of light sources for a given spatialangle, such that multiple views are provided. Note that while spatialmultiplexing is thus generally based on a mechanical design of arrayedlight sources and lenses, the design also influences how temporalmultiplexing works in conjunction with the spatial multiplexing, anddetermines the range of applications for which the combination ofspatial and temporal multiplexing is suited.

While the capabilities of the spatial multiplexing are largely fixedbecause they are due to a fixed physical construction (e.g., alenticular array), temporal multiplexing is inherently dynamic and thuscan be configured to select varying patterns of illumination over time,supporting varying kinds of applications, depending on the programmingof the directed backlight pattern.

For example, by both temporally multiplexing (using a collimateddirected backlight) to provide X views based on the frame ratecapabilities of the display device and spatially multiplexing (e.g.,using a parallax barrier/lenticular display or a lenslet displayaccording to the spatial capabilities of the display device) to provideY views, up to X times Y views may be generated for a given display.When used with a head tracking system, these views may each be steeredto the current and/or expected location of an eye to maintain arelatively high resolution for each of the views. As a more particularexample, given a 480 hertz display (which presently exist), it ispossible to generate four autostereoscopic views (3D views withoutglasses, accomplished by directing a different view to each eye) usingtemporal multiplexing alone, each view receiving frames at 120 hertz. Byincluding spatially multiplexing functionality capable of generatingfour views, for example, the display is able to generate sixteendifferent views. Note that in this example seven more views are providedthan the nine-view lenticular display provides, yet the horizontalresolution not as degraded, that is, only by one-fourth compared toone-ninth.

It should be understood that as used herein, a “view” may refer to anindividual eye of a person, such as when displaying stereoscopic (3D)video; alternatively, a view may be the same to both of a person's eyes.Thus, the sixteen views in the above example may be generated to showpersonal stereoscopic video to eight viewers, one different view to eacheye. Note that a “personal” view means that one viewer may be receivingdifferent video images relative to the images received by anotherviewer, such as for viewing a view of the same scene as seen by anotherviewer, but from a different angle, for example, or for “simultaneously”(but possibly time multiplexed) watching different television shows onthe same display. Further note that if there are more eyes thanavailable views, some or all of the views may be used for outputting 2Dvideo, and/or non-personal 3D video, as generally described in theaforementioned related U.S. patent application entitled “Optimization ofa Multi-View Display.”

One other advantage of combining spatial and temporal multiplexing isthat a known shortcoming of lenticular autostereoscopic displays may beremedied. More particularly, parallax barrier techniques generate a setof fixed views that repeat in a regular fashion over the whole viewingangle of the display. For example, as the user moves from left to right,a parallax barrier may generate views ranging sequentially from viewone, to view two, to view three. As the user moves further right, thedisplay returns to view one, then to view two, then to view three and soon.

This repeating nature of the views is a problem in scenarios where it isdesirable to present unique views throughout the viewing range of thedisplay. For example, it may be desirable to present unique views tomultiple people, such as in one possible scenario in which a simulatedviewing experience is that two people are looking at the same 3D object,but with correct parallax effects. That is, the two people see theobject at the same 3D position (at the same room coordinates), howevereach can see a part of the object that the other cannot, correspondingto a correct viewing angle; their views are as if they are both lookingat an object in the room from different angles, rather than the usualautostereoscopic experience where both users have precisely the sameviewing experience. For example, one person may see the front of a box,while the other sees the side. As the person seeing the front moves to adifferent viewing angle (e.g., view two), that person sees part of theside of the box as well. With parallax barrier techniques/repeatingviews, as the person moves too far (e.g., past view three), instead ofseeing the box from a corresponding angle, with only spatialmultiplexing, the view jumps back to the initial front-of-the box view.

Another advantage of the combination of spatial and temporalmultiplexing is that the demands on each technique may be lessened whilestill maintaining a useful number of views. As mentioned above, thehighest contemporary lenticular display enables nine separate views, butthis results in significantly degraded resolution. By employing temporalmultiplexing, the lenticular display may be designed to support fewerspatial views, such as three, thereby reducing the adverse effect onresolution. However, in the above example, time multiplexing via thedirected backlight can still provide (at least) the nine views at anacceptable frame rate.

Turning to another aspect, dynamically driving the temporal multiplexingfacilitates exploiting various information that allows determination ofwhen and how to enable a particular view by the directed backlight. Forexample, if the system is used with face or eye tracking technology, thesystem can only enable views that would fall on the users' eyes, thuspotentially increasing frame rate and/or brightness, and/or savingenergy.

By way of example, consider a 240 hertz display with three spatiallymultiplexed views, providing up to twelve views at 60 hertz each. If asingle person is viewing the display, significant energy may be saved byshowing only one view to that user. However, user preference data may beaccessed to determine how to trade off desired quality versus energysaving, e.g., a user may want higher resolution and a higher framerate/brightness than that provided if the maximum number of views werebeing generated. In general, the dynamic aspect of the temporalmultiplexing allows the ordering and scheduling of views in a dynamicfashion to support efficient illumination strategies that preserveresolutions and brightness.

Note that the views shown to different people need not have the sameactual or perceived quality. For example, one person may get a higherframe rate than another, at least for awhile, so as to not noticeablydegrade one person's viewing experience just because another personhappens to be walking through room and glances at the display. A personwatching a show with significant motion (e.g., as determinable from MPEGmotion information) such as a sporting event may get a higher frame ratethan a person watching the same display at the same time to view aprogram will less motion, such as a drama. Thus, for example, a personwatching a show with a lot of motion may receive more time slices (e.g.,corresponding to a higher frame rate) than another person watching ashow with less motion.

FIG. 2 is a block diagram showing an example implementation thatcombines spatial and temporal multiplexing to provide a plurality of(typically) differing views. The various components of FIG. 2 may beimplemented in computing environment external to a display, e.g., in acomputer system or gaming console, for example, containing one or moreprocessors and memory to perform spatial and temporal multiplexing.Alternatively, the components may be incorporated into a display device,such as a television set.

In the example of FIG. 2, an eye (head) tracking component 202 (e.g.,using one or more cameras and software technology) provides adistribution mechanism/distribution logic 204 with the current number ofviews (eyes) and the initial position of each eye. Note that such headtracking already exists, based on known machine vision softwaretechnology, and may be incorporated into gaming consoles, for example.Notwithstanding, any eye tracking technology may be used as describedherein, including technology not yet developed.

Note that the video signals may be anything, such as a single videoshot, a mix of different angles of the same 3D scene, a mix of differentvideo shots (e.g., two television programs to distribute betweendifferent viewers) and so forth. Thus, other data that may be needed inassociation with each view is a content identifier. In general, ifdifferent people are currently using the multi-view display to viewdifferent video clips (e.g., television shows), there needs to be anassociation between each view and the content to show to that view. Forexample, a user can make a head gesture or other gesture so that thehead tracking component can associate the person's eyes with a position,in conjunction with that person using a remote control to indicate theshow that the user wants to see. Thereafter, the person sees that showuntil otherwise changed.

Note that if all views are of the same 3D scene but some (or all) viewscorrespond to different viewing angles, the content to show each viewmay be known based on the position. In such a scenario, the positiondata of each view serves as the “content” identifier, namely the angleat which the content is viewed.

As described herein, the distribution logic 204 determines how todistribute video signals 206 among the views via spatial and temporalmultiplexing. Preference data 208 (which may be fixed defaults for agiven display device) may be accessed to determine how the distributionproceeds, e.g., how to correlate more views than can be output given thespatial and frame rate capabilities of the device, or conversely, whenless than all possible views are present, how to increase resolution andbrightness, and/or to which view or views, and so forth. Note that thedistribution logic 204 also may input (data corresponding to) the videosignals, such as to determine whether the video is 2D or 3D, and also topossibly process motion information in determining the distribution.

The distribution logic 204 provides the computed parameters (e.g., setsof a view number, initial position, timing data, and content identifier)to a scheduling algorithm 210. In general, the scheduling algorithm 210outputs spatial and temporally multiplexed signals to graphics hardware212 to control which drivers 106 illuminate which LEDs/LCDs and when,thereby providing a multi-view display 214. This may includenon-illuminated frames to views where there is no viewer present, tosave energy.

FIG. 3 shows some example steps that may be performed by thedistribution logic 204, beginning at step 302 where the number of eyes(maximum possible views desired) is received from the head trackingcomponent. At step 304, the distribution logic 204 computes how manyviews are needed based upon the number of views and the 2D and/or 3Dnature of the video signals 206. As described above, the preference data208 is consulted to determine what to do when there is an inexact matchbetween the views needed versus those that can be independentlygenerated, e.g., which view or views are to be degraded (e.g., combined)or enhanced according to the preference data 208, given the currentnumber of possible views. Content-based throttling or enhancement mayalso be performed, e.g., based on how the preference data specifies thatcurrent motion information be used, as described above.

In general, the scheduling algorithm 210 receives parameters from thedistribution logic 204, e.g., a set of numbered views, the position thatthe view is currently at, the timing data for that view (e.g., if it isnot equally divided for every view), and what content to display to thatview, if needed.

Given the parameters and the position information from the head trackingcomponent, the scheduling algorithm 210 computes the spatial andtemporal multiplexed output for each view, that is, which light sourcesto illuminate based on the view's frame rate to show a video frame to aparticular view's current position. Note that in this example thescheduling algorithm 210 receives the position data and re-computeswhich light sources to illuminate for a view's viewing time as eachperson moves, and thus the distribution logic 204 does not need tore-compute the parameters each time a person moves. However in thisexample, the distribution logic 204 re-computes the parameters whenthere is a state change with respect to the number of views, e.g., oneviewer leaves or another one joins, and so forth. State change isfurther described in the aforementioned related U.S. patent applicationentitled “Optimization of a Multi-View Display.”

CONCLUSION

While the invention is susceptible to various modifications andalternative constructions, certain illustrated embodiments thereof areshown in the drawings and have been described above in detail. It shouldbe understood, however, that there is no intention to limit theinvention to the specific forms disclosed, but on the contrary, theintention is to cover all modifications, alternative constructions, andequivalents falling within the spirit and scope of the invention.

What is claimed is:
 1. A multi-view display comprising: multiple lightsources configured to generate light; a parallax barrier or lensletconfigured to direct the light to different directions depending onwhich of the multiple light sources are illuminated; and a schedulingalgorithm configured to direct the light to different viewing angles byselectively illuminating different light sources behind the parallaxbarrier or lenslet in different time slices, wherein the differentviewing angles correspond to different views, wherein the differentviews include a first view that is directed to a first viewing angle byilluminating a first light source behind the parallax barrier or lensletduring a first time slice and a second view that is directed to a secondviewing angle by illuminating a second light source behind the parallaxbarrier or lenslet during a second time slice, wherein the first viewingangle is different than the second viewing angle.
 2. The multi-viewdisplay of claim 1 wherein the different views includes a third viewdirected toward a first eye of a third person and a fourth view directedto a second eye of the third person.
 3. The multi-view display of claim1, further comprising: an eye or head tracking component configured totrack eye or head positions of a first person and a second person,wherein the scheduling algorithm is configured to receive position datafrom the eye or head tracking component and to redirect one or more ofthe different views based on the position data.
 4. The multi-viewdisplay of claim 3, wherein the eye or head tracking component comprisesone or more cameras.
 5. The multi-view display of claim 1, wherein thedifferent views correspond to at least two different television programsdisplayed concurrently on the multi-view display.
 6. The multi-viewdisplay of claim 1, further comprising: a distribution mechanismconfigured to compute timing parameters based at least in part on adesired number of views, wherein the scheduling algorithm is configuredto use the timing parameters to generate the different views.
 7. Themulti-view display of claim 6, wherein the distribution mechanismcomputes the timing parameters for the different views based uponcorresponding content.
 8. The multi-view display of claim 6, wherein thedistribution mechanism is configured to compute timing or resolutionparameters, or both timing and resolution parameters, based upon anumber of the different views that are generated.
 9. The multi-viewdisplay of claim 6, wherein the distribution mechanism is configured tocompute timing or resolution parameters, or both timing and resolutionparameters, based upon preference data.
 10. The multi-view display ofclaim 1, wherein the multiple light sources comprise multiple lightemitting diodes or multiple liquid-crystal displays.
 11. The multi-viewdisplay of claim 1, wherein the multi-view display has a given rating inhertz and different views are provided at hertz rates that are divisorsof the given rating.
 12. The multi-view display of claim 11, wherein thegiven rating is 480 hertz and the divisors are 120 hertz.
 13. A systemcomprising the multi-view display of claim 1 and a computing environmentcomprising a hardware processor configured to execute the schedulingalgorithm.
 14. The system of claim 13, wherein the computing environmentcomprises a gaming console.
 15. The system of claim 13, embodied as atelevision set.
 16. The multi-view display of claim 1, wherein thescheduling algorithm is further configured to concurrently displaydifferent shows to different viewers at different frame rates.
 17. Themulti-view display of claim 16, wherein the different shows displayedconcurrently include a sporting event and a drama, and the schedulingalgorithm is configured to concurrently show the sporting event at ahigher frame rate than the drama.
 18. The multi-view display of claim 1having the lenslet, wherein the lenslet and the multiple light sourcesare positioned such that the lenslet directs the light to the first viewwhen the first light source is illuminated and the lenslet directs thelight to the second view when the second light source is illuminated.19. The multi-view display of claim 1 having the lenslet, wherein thelenslet and the multiple light sources are positioned such that thelenslet directs the light to the first view and not the second view whenthe first light source is illuminated and the lenslet directs the lightto the second view and not the first view when the second light sourceis illuminated.
 20. The multi-view display of claim 1, wherein thescheduling algorithm is configured to selectively illuminate the lightsources such that a first portion of an object is shown to a firstperson during a first time slice and a second portion of the object isshown to a second person during a second time slice, wherein the firstportion of the object shown during the first time slice is not visibleto the second person and the second portion of the object shown duringthe second time slice is not visible to the first person.